Type I Error Rates and Power of Multiple Hypothesis Testing Procedures in Factorial ANOVA

There are numerous General Linear Model (GLM) statistical designs that may require multiple hypothesis testing (MHT) procedures that control the Type I error inflation that occurs with multiple tests. This study investigated familywise error rates (FWER) and statistical power rates of several alpha-adjustment MHT procedures in factorial ANOVA, but results apply broadly to GLM procedures. Of four MHT procedures investigated, the Hochberg procedure performed most efficiently in terms of Type I error and power, slightly better than Holm. The Holm procedure, however, may be the better choice because of less restrictive assumptions. FWER concerns were raised with the Benjamini-Hochberg procedure. n educational research, many statistical analyses are conducted using null hypothesis significance tests. Often, only a single null hypothesis is tested. For example, an investigator might test whether a group mean differs from a specified value or if there is any significant difference between an intervention and a control. To answer these research questions, various types of t tests could be adopted under a given level of significance, or α. In statistics, if the null hypothesis is incorrectly rejected, Type I error occurs. That is, there is a 5% of chance that researchers can make a Type I error for a single hypothesis test when α = .05. However, when k multiple hypotheses are tested, the k separate null hypothesis significance tests are often performed each at α = .05. The probability of making at least one Type I error when multiple, independent true null hypotheses are tested is defined as p=1−(1− α) k , where α is the nominal Type I error rate (often .05) and k is the number of independent hypothesis tests performed (Hochberg & Tamhane, 1987; Maxwell & Delaney, 2000; Schochet, 2008; Stevens, 2002; Toothaker, 1993). For example, if there are four independent tests (k = 4), the probability of finding at least one spurious impact is .19, .40 for 10 tests, and .87 for 40 tests (Schochet). Therefore, the Type I error rate is inflated across multiple hypothesis tests (MHT). Many researchers are familiar with post hoc multiple comparison procedures, which are able to maintain the familywise Type I error rate at α when a set of post hoc comparisons is made among sample means, following a significant one-way ANOVA. These procedures (e.g., Tukey; Scheffé) adjust nominal alpha for each test so that the probability that at least one of the significant null hypotheses is a Type I error remains at a given familywise alpha. However, multiple comparison procedures (MCPs) tested in one-way ANOVA represent only one area that uses MHT. In fact, MHT can be conducted in several other areas, such as (a) multiple tests of correlations among variables, (b) univariate and multivariate post hoc tests in MANOVA methods, (c) multiple chi-square tests in differential item functioning, (d) tests of multiple coefficients in multiple regression, (e) repeated measures post hoc tests, and (f) tests of multiple sources of variation in factorial ANOVA. Several kinds of general purpose MHT procedures have been made available for researchers, including the Bonferroni procedure, the Holm (1979) procedure, the Hochberg (1988) procedure, and the Benjamini-Hochberg (1995) procedure. Like MCP methods, the use of general MHT procedures is to control the Type I error rate when testing multiple hypotheses simultaneously. Several scholars have suggested that the selection of MHT procedures is mainly based on the Type I error rate and statistical power (Hochberg & Benjamini, 1990; Kirk, 1995). As Kirk stated, "Other things equal, a researcher wants to use a procedure that both controls the Type I error rate at an acceptable level and provides maximum power" (p. 123). As Keren and Lewis (1993) stated, "Ideally, we would like to select a method that provides a powerful test while maintaining adequate Type I error control, requires few statistical assumptions, and is easy to apply" (p. 56). Therefore, Type I error rates and statistical power rates were investigated to evaluate several MHT procedures in this study.